Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: waiwai933 Date: Nov 22, 2008 4:10pm
Forum: web Subject: Knowing if a page has been crawled

Is there any way to know if a page has been crawled in the past but just not updated in the archives?

Reply to this post
Reply [edit]

Poster: kustota Date: Nov 22, 2008 7:10pm
Forum: web Subject: Re: Knowing if a page has been crawled

http://en.wikipedia.org/wiki/Web_crawler#Crawler_identification